Data Engineer (Hadoop/Spark) | 6-Month Contract

Responsibilities:

Data Pipeline & Architecture Development

• Build and optimize automated data pipelines for ingesting, transforming, and processing large datasets.
• Design efficient data architectures that support analytics, machine learning, and real-time applications.

Cloud Migration & AI Enablement

• Support cloud migration efforts, transitioning on-premises data workflows to cloud-based platforms like Databricks.
• Collaborate with data scientists to improve feature selection, feature engineering, and enable end-to-end AI workflows from model training to deployment and monitoring.

CI/CD & Automation

• Develop CI/CD pipelines to streamline data pipeline deployments and ensure stable, automated workflows.
• Improve monitoring and observability to maintain system reliability.

Collaboration & Business Impact

• Work with data scientists, product teams, and platform engineers to align data solutions with business objectives.
• Ensure data quality, security, and compliance with industry standards.
• Contribute to best practices in data governance, documentation, and automation.

Qualifications & Skills

• Degree holder of Information Technology, Mathematics, or Statistics with at least 1-2 years of experience in data engineering.
• Hands-on experience with Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), Spring Boot, etc., to build big data products & platforms.
• Expertise in Python or Java.
• Experience in UNIX environment, Git Flow, CI/CD automation, Jenkins, Bitbucket.
• Proficiency with a modern cloud or hybrid-cloud stack (AWS, Databricks, Cloudera, etc.).
• Experience in building and deploying production-level data-driven applications and data processing workflows or pipelines.